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CM Process Assessment Strategy 

“Introduction” 


Purpose: To propose a strategy for assessing the development and 
effectiveness of configuration management systems within Programs, 
Projects, and Design Activities performed by technical organizations and 
their supporting development contractors. 

Scope: Various entities CM Systems will be assessed dependent on Project 
Scope (DDT&E), Support Services and Acquisition Agreements. 

Approach: Model based structured against assessing organizations CM 
requirements including best practices maturity criteria. The model is tailored 
to the entity being assessed dependent on their CM system. 



CM Process Assessment Strategy 

“Why Assessments vs. Audits” 


The assessment approach provides objective feedback to Engineering and Project 
Management of the observed CM system maturity state versus the ideal state of the 
configuration management processes and outcomes (system). 

Identifies strengths and risks versus audit gotcha' s (findings/ observations). 

Used "recursively and iteratively" throughout program lifecycle at select points of 
need. (Typical assessments timing is Post PDR/Post CDR) 

Ideal state criteria and maturity targets are reviewed with the assessed entity prior 
to an assessment (Tailoring) and is dependent on the assessed phase of the CM 
system. 

• Supports exit success criteria for Preliminary and Critical Design Reviews. 

• Gives a comprehensive CM system assessment which ultimately supports 
configuration verification activities.* 



CM Process Assessment Strategy 


“CM Process Audits (Assessments) are an important contributor towards Configuration Verification 
and ultimately acceptance of design solutions and products (Cl’s).” 


Configuration Verification Audits* 

Inputs 

Mech./Facilitators 

Constraints Output 

Results 

• Audit schedules, agendas 
and requirements 

• Other Configuration 
verification results and 
action items 

• Status and configuration 
information from the CSA 
system 

• Physical Cl (hardware and 
software) test results 

• Documented CM Processes 

• Open communication 

• Manufacturing and 

• Engineering Tools 
documentation 

• Contractual provisions 

• Audit criteria contained in 
the Audit Plan 

• Audit Report that 
includes results, findings, 
certification package(s) and 
action items 


The Configuration Verification activity that establishes that: 

■ The product meets requirements-performance and functional requirements defined in the 
product definition information have been achieved by the design. 

■ The documentation matches the product-design has been accurately documented in the 
product definition information. 


‘Adopted from DAU Configuration Verification Training Module 
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OACDM 


Characteristics of the Maturity Levels 

(Adopted from SE-CMM Version 1.1) 


Level 1 
Performed 
Informally 


r 


j 


Level 2 
Planned & 
Tracked 




J 


"\ 


Level 3 

Well 

Defined 




"A 


Level 4 
Quantitatively 
Controlled 


J 


Level 5 
Continuously 
Improving 



Processes 
unpredictable. 
Characterized by 
individual 
knowledge and 
effort. 


Processes 
defined. 
Characterized 
by standards . 


Processes 

refined. 

Characterized by 
tailored standard 
processes. 


Processes 
measured & 
controlled. 
Characterized by 
quantitative 
understanding. 


Processes 

proactive. 

Characterized 

by 

quantitative 

goals. 


— Maturity Statement Levels are tailored through consensus including “not applicable” levels before an entity is assessed — 






CM Process Assessment Strategy 

“Assessment Modules Description” 


The Current Model Construct to be used for Assessments is based on the five 
CM functions recognized in Industry and Documentation Requirements and 
System Descriptions (Characterized as Maturity Statements). Each module 
contains the applicable CM requirements mapped to the CM functional 
module they apply to and include supporting maturity criteria to support the 
assessment: 

■ Configuration Management Planning Module (2 Reqs) 

■ Configuration Identification Module (9 Reqs) 

■ Configuration Change Management Module (11 Reqs) 

■ Configuration Status and Accounting Module (4 Reqs) 

■ Configuration Verification Module (1 Req) 


Assessment Model Considerations 

• Processes can overlap CM tenets 

• Does not include SCM or TDM 

• Requirements are mapped to module criteria 

• Criteria are developed from best practices and mapped to the 
applicable requirements 



CM Process Assessment Strategy 
“Assessment Model Description” 


Expected Maturity Levels for each essential configuration elements were 
derived from Program Configuration and Data Management Best 
Practices and Audit Plans. 


Assessment Modules 1 

Target Maturity 
Level PDR 
Assessment 

Configuration Management j 

Level 3-4 

Planning • 

1 

Configuration Identification | 

Level 3-4 

Configuration Change 1 

Level 2-3 

Management 


Configuration Status Accounting I 

Level 2-3 


Target Maturity 
Level CDR 
Assessment 


Level 5 
Level 4 
Level 4 


Level 4 


r 

► 


s The outcome is characterized and reported at the module level as a set of 
assessment outcomes for maturity and candidate risks. 
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“Assessment Model Reference Documentation” 


Model Approach, Requirements, and Criteria within each configuration element 
module were derived from the following sources: 

s Assessment Model Development 

■ AIA NAS3500 National Aerospace Standard 3500 Technical Data Package: Composition, Communication, and 
Application (Model and Risk Outcome Scheme) 

■ Systems Engineering Capability Maturity Model (SE-CMM registered service mark of Carnegie Mellon University). 
(Used for assessment levels) 

a Program Configuration and Data Management Audit and Verification Plan 

0 CM Requirements (Beta Test) 

■ Systems Engineering Processes and Requirements 

a Data Requirements Document Standard for a CM Plan 

■ Engineering Documentation Standards 

■ Computer - Aided Design (CAD) Standards 

■ Configuration Management Standard 

0 Assessment Criteria 

■ Engineering Documentation Standards 

■ Computer - Aided Design (CAD) Standards 

■ Configuration and Data Management Audit and Verification Plan 

• Configuration Management Guidance 

• Configuration Management Standard 
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Assessment Process Overview 

Preliminary Assessment Planning 

Identify the entity (s) to be assessed (e.g. program, project, organization, element). 

Define the assessed entity(s) organizational structure and the personnel needed to support the Assessment. 

0 Define scope (depth and breadth) of the review. 

0 Develop and document preliminary assessment approach based on above. 

0 Create Assessment In-Briefing 

Conduct Initial Assessment Activities 

Begin collection of reference material for each activity subject to assessment and begin the entity assessment: 

0 Document any considerations of "tailoring" and any sampling scheme. 

0 Compile and organize collected reference material. 

0 Develop the initial tailored model for the assessment. 

Conduct Pre Assessment Meeting 

Meet with Entity Management to explain and obtain consensus of assessment objectives, methodologies, and tailored models. 

0 Establish consensus on the breadth and depth of the assessment (i.e. government/ contractor). 

0 Review resources required to perform the assessments and how they will be provided. 

0 Review the assessment schedule. 

Initiate Primary Assessment 

0 Based on reference material prepare assessment models. 

0 Schedule and hold meetings with assessed organization POCs and collect supporting objective evidence. 

0 Evaluate collected evidence against assessment criteria and coordinate with entity personnel as needed. 

Prepare Outcome Reports 

0 Complete narratives and scores of assessment models, candidate risks, and results summaries. Generate out brief presentation and draft 
report. 

0 Present results to assessed entity and CM management 
0 Release final report. 






Supporting Process 
Documents: 

Program/Project Plans (including CM 
and SEMP) 

CCB Charters 

Program/Project Drawing and Cl/Spec 
Trees, IPLs. 

Change Management Artifacts 
(Sub-Process Ol’s, Change Files, CSA 
Reports, etc.) 


DOCUMENTS AND PRODUCTS 
AFFECTED BY THIS CHANGE: 

Program-To-Project Systems ICD, 

Core Stage-to-Project Detailed Design Data 


Project Vendor’s 
Configuration Data 
Affected by this Change: 


External Assessment Activity 


APPLICABLE DOCUMENTS: 


Cross Program Fluid Procurement and Use Control Specification; 
Program to Project ICD, Volume 1: Functional Interface Definition & 
Program Integrated Vehicle to Project Detailed Design 
Program-to-Project ICD Volume 5: Program-to-Project Command and 
Data Handling (C&DH) Detailed Design 
Cross Program Integrated Vehicle Coordinate Systems 
Program System Specification 


APPLICABLE DRAWINGS: 

Umbilical Electrical Connector Envelope Drawing; 

Vent/ Relief - Fill/ Drain Quick Disconnect (QD) Envelope Drawing; 
Core Stage TVC Hydraulic Ground Service Panel QD Source Control 
Document (SCD); Quick Disconnect Assembly, ECS Purge, Ground 
Helium Quick Disconnects Envelope Drawing; Main Propulsion 
System, Core Stage LOX and LH2 Bleed Disconnects Envelope 
Drawing; Main Propulsion System, Core Stage Drawing 
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“Notional Assessment Model Excerpt” 


Configuration Element 
CM Planning 

Assessment Notation 

Project 

Milestone; 




Expected 

Actual 

Potential Risk 

PI. A strategy shall be 
prepared to conduct 
configuration 
management for 
configuration items 
and configuration 
documentation. (MPR 
7123.1 2.214.1a) 

PDR Expected: Level 3 

CDR Expected: Level 5 

By this time in the development process the Configuration Element CM 
Planning should be a Maturity Level 3 Well Defined. This maturity level 
means refined processes characterized by tailored standard processes. 

It was determined during this assessment that key criteria of [1) 
Organizational relationship, [2) Staffing, and (3) Training were at 
Maturity Level 2 Planned & Tracked, In each criterion the processes 
were defined and characterized by standards but they were not refined 
by tailoring to the circumstances. The Level 2 for such a number of 
criteria result in a Level 2 ranking for the entire element. 

3 

2 

There is a risk that expectations of 
the Program Office will differ with 
expectations of CDM Office, 

Lack of defined organizational 
relationship results in risk of 
ineffective assignment of person nel. 
Inadequate staffing and training 
results in risk of ineffective 
performance of key CM functions. 

PI Criteria 

Assessment Notation 


CM Organizational 
charts and functional 
job descriptions are 
available.(CxP 70073- 
01-03 Appendix C) 

Organizational Charts and Functional fob Descriptions were provided when requested 
and are maintained as part of a document maintained by the Program CDM Lead. 
Organizational Charts and Job Descriptions were adequately detailed and current. [Ref: 
WP Pin 100) WD. NR. 


Or ga ni zati on al 
relationship of CM/DM 
Office to program 
clearly defined and 
documented. (MGM 
8040,1 Table III) 

Other than general references to high-level policy criteria that CM/DM Office shall 
provide "support" for applicable programs there is not a documented and clearly defined 
organizational relationship. The CDM Lead stated some of the CDM personnel assigned 
non CDM tasks by Program management. The CDM Lead stated so far the unrelated 
assignments have not yet been significantly disruptive but are likely to become so as 
work load increases with maturing program. P&T. CR. 

Lack of documented relationship 
fosters a risk of ineffective 
assignment of CM personnel at the 
expense of core CM duties. 

All applicable areas of 
CM are staffed. [MGM 
8040.1 Table III) 

Applicable areas of CM are staffed with the exception of Board Secretariat. Currently 
[and for the past 18 months) the duties of the vacant position have been rotated among 
available CM personnel. The CDM Lead stated the duties of the Board Secretariat have 
been performed but with some confusion and a slower pace than desired, P&T. CR 

inadequate staffing of a core CM 
position results in risk of ineffective 
performance of key CM duties. 
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‘Final Report Content’ 





Section 1 

Summary of the Assessment Activities including the General Process with any 
Unique Assessment Process Outcomes and a Summary Statement for each 
Assessment Module Result. 

Section 2 

The Final Tailored Assessment Model with all Assessment Notes. 

Section 3 

The Candidate Risks List Derived from the Assessment Outcome mapped to the 
relevant Modules, Requirements and Criteria Statements. 


Note-The final report draft is reviewed with the assessed entity to obtain consensus on the results before 
the final report is released. 



